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Introduction 


he Math in Common (MiC) initiative 

was launched in 2013, amid the intro- 
duction of many education policy changes 
in California. The California State Board of 
Education adopted the Common Core State 
Standards for Mathematics (CCSS-M) in 2010, 
although there was a delay in associated state 
policy supports for CCSS-M implementation. 
For instance, there was no state-approved list 
of CCSS-M-—aligned instructional materials 
until 2014, and the first standards-aligned 
summative achievement test was not admin- 
istered until spring 2015. 


The MiC initiative aimed to support 

10 California districts in implementing 

the CCSS-M and improving mathematics 
teaching and learning in grades K—-8. Another 
goal of the initiative was for participating 
districts to identify and share best practices 
that could help the state’s other 900-plus 
districts accelerate implementation of the 
CCSS-M and improve their math achieve- 
ment (S.D. Bechtel, Jr. Foundation, 2012). 


Five years into the initiative, and with sev- 
eral years of data available from the state's 
standards-aligned summative achievement 
test, the California Assessment of Student 
Performance and Progress (CAASPP), we 
are in a position to examine some trends in 
student achievement scores in the CCSS-M 
era. Analyzing these data can help us under- 
stand how MiC districts, with their infusion of 
both material and intellectual resources, are 
performing in relation to the state math- 
ematics standards, and how this progress 


looks when compared to peer districts and to 
districts across the entire state. 


This report documents the uneven math 
gains made by MiC districts, schools, and 
students, by analyzing the patterns of those 
gains, which range from outperforming 
statewide trends to more moderate growth. 
While progress in student achievement in the 
MiC districts has been slow, there are some 
promising signs to share with the field. 


Slow improvement in 
California mathematics 
achievement 


In California, student achievement outcomes 
in mathematics have been disappointing 

to educators, parents, and students (Cano, 
2018; Reardon et al., 2018). CAASPP math- 
ematics data reveal that the progress of 
student achievement in mathematics since 
2015, the first year in which the CAASPP was 
administered, is slow and lags behind student 
achievement in English language arts (see 
Tables 1 and 2). Educators and observers 
want to see higher percentages of California 
students meeting or exceeding the stan- 
dard as set forth in the CAASPP,? but those 
percentages are mostly stagnant. 


Tables 1 and 2 show statewide CAASPP 
performance and progress data for both 
mathematics and English language arts 

for two student cohorts across years of 
administration: an elementary school cohort 


1 Mathematics scale scores (which are students’ raw test scores converted into a common scale that is comparable 
across grades) increase as students progress across grade levels, as would be expected given the psychometric proper- 


ties of the assessment (Warren, 2018). 


2 Four achievement levels are specified for the CAASPP: standard not met, standard nearly met, standard met, and 


standard exceeded. 


Table 1. California Student CAASPP Performance for Two Student Cross-Grade Cohorts, 2015-2018 


Average | Average | Average | Average | Scale Scale Scale 
Scale Scale Scale Scale Score Score Score 
Score, Score, Score, Score, Increase, | Increase, | Increase, 
2015 2016 2017 2018* 2015- 2016- 2017- 
2016 2017 2018 
Elementary school cohort 2415.1 2460.5 2485.8 2511.0 45.4 25.3 25.2 
Math 
Elementary school cohort 2402.9 2454.4 2489.5 2518.9 51-4. 35.1 29.4 
English language arts 
Middle school cohort 2504.4 2525.0 2540.2 N/A* 20.6 15.2 N/A* 
Math 
Middle school cohort 2511.8 2542.0 2559.0 N/A* 30.2 17.0 N/A* 
English language arts 


* 2018 data for the grade 6 cohort are not available because the CAASPP is not administered to grade 9 students. 


Source: California Department of Education, 2018a. 


Table 2. California Student CAASPP Achievement Progress for Two Student Cross-Grade Cohorts, 


2015-2018 

Percentage Percentage Percentage Percentage 
Meeting/ Meeting/ Meeting/ Meeting/ 
Exceeding Exceeding Exceeding Exceeding 
Standard, 2015 Standard, 2016 Standard, 2017 Standard, 2018 

Elementary school cohort 40 38 34 37 

Math 

Elementary school cohort 38 44 47 48 

English language arts 

Middle school cohort 33 36 36 N/A* 

Math 

Middle school cohort 42 48 49 N/A* 


English language arts 


* 2018 data for the grade 6 cohort are not available because the CAASPP is not administered to grade 9 students. 


Source: California Department of Education, 2018a. 


(who were tested in grade 3 in 2015,3 in cohorts’ average math scale scores increased 
grade 4 in 2016, and in grade 5 in 2017) and each year. However, Table 2 shows that 

a middle school cohort (who were tested the percentages of these same students 

in grade 64 in 2015, in grade 7 in 2016, and meeting or exceeding the standard in math 
in grade 8 in 2017). Table 1 shows that both were stagnant. 


3 CAASPP assessments are administered in the spring of each school year. For the purposes of this report, the assess- 
ment date (e.g., 2015) indicates the spring of the school year (e.g., 2014-15) in which the assessment was administered. 


4 In some California school districts, grade 6 is included in the elementary level. For the purposes of this report, grade 6 
is included as the baseline year for the middle school cohort of students. 
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While, as shown in Tables 1 and 2, mathe- 
matics achievement levels are disappointing 
across all California students, achievement 
data for three subpopulations of California 
students — English learners, students with 
disabilities, and low-income students — have 
been even more disappointing (Cano, 2018; 
Education Trust—West, 2018a; Reardon 

et al., 2018; Warren, 2018). For example, the 
Education Trust—West (2018a) summarized 
the particularly poor performance of English 
learners on the most recent (2018) CAASPP 
mathematics and English language arts tests, 
noting that English learners’ 2018 scores 
increased by less than 1 percentage point 
over the 2017 results. In a related report, 
Education Trust—West (2018b) researchers 
lauded the efforts of California policymakers 
to support equity for English learners — 
including the state’s adoption of the English 
Language Development Standards in 2012 
and the English Learner Roadmap in 2017 

— but noted that there is more work to do 
to “dismantle the inequities far too many of 
our students face” (Education Trust—West, 
2018b). 


Although state policymakers have assured 
Californians that student achievement is 
expected to improve over time (California 
Department of Education, 2018b), in the short 
term, the public is apt to wonder why mathe- 
matics achievement, as measured by state- 
wide assessments, is not improving. Recent 
research from Policy Analysis for California 
Education and Stanford University (2018) has 
identified a host of issues (both inside and 
outside of California's education system) that 
may be contributing to both the overall disap- 
pointing mathematics results and disparities 
among different student subgroups, including 
financing that creates disparities between 
poorer and wealthier communities; early 
childhood education and poverty; and the 
capacities of teachers and principals. 


Of course, CAASPP data and data from other 
high-stakes student assessments cannot, 

on their own, offer a complete picture of 
either student achievement in mathematics 


in California or the state’s progress in imple- 
menting the CCSS-M. Based on years of 
work with the MiC initiative, we know how 
complicated it can be to accurately measure 
the deep changes that are occurring in 
mathematics classrooms across the initiative 
and across the state. While assessment data 
offer critical insights into how students are 
performing, it also generally makes sense 

to look beyond student test scores to gain a 
fuller understanding of how CCSS-M imple- 
mentation is playing out in classrooms. For 
instance, conducting classroom observations 
and analyzing the resulting data can provide 
useful on-the-ground insights about math 
teaching and learning (Chu, Perry, Reade, & 
Marple, 2019). 


Math in Common 

districts: Priorities, 
demographics, and CCSS-M 
implementation timelines 


Understanding the MiC districts’ priorities, 
demographics, and implementation timelines 
is important in order to contextualize the 
tables, figures, and analyses offered in this 
report. 


Focusing on equity. In 2012, the S.D. Bechtel, 
Jr. Foundation invited 27 California districts 
whose students were underperforming on 
state mathematics achievement measures 
to apply for MiC funding; 23 districts sub- 
mitted proposals to participate, and 10 were 
accepted into the initiative. One criterion for 
districts to be invited to submit a proposal 
was the extent to which district leaders were 
already working to close achievement gaps 
between their student populations and the 
statewide student population. All 27 of the 
invited districts already had student equity as 
a focus of their work. 


In order to understand how the eventual 

10 MiC districts’ work toward achieving 
equity among their students has played out in 
testing, this report analyzes achievement for 
all students in each district, as well as for one 
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Figure 1. CCSS-M Implementation Timelines for the Math in Common Districts 
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Source: Math in Common district leadership team members. 


student subgroup — English learners — in 
each district.> 


As a comparison group for the MiC districts, 
we analyzed student scores from the 13 dis- 
tricts® that applied for, but did not receive, 
MiC funding. We used these districts as a 
comparison group because, like the 10 MiC 
districts, they all had plans for creating 
equitable learning environments supportive 
of all students, so we have assumed that they 
continued to pursue student equity without 
the extra supports that MiC provided. 


MiC district demographics. At the beginning 
of the initiative, the percentages of English 
learners in the 10 MiC districts ranged 

from 16 percent (Elk Grove) to 82 percent 
(Sacramento City), with an average English 
learner percentage of 43 percent across the 
10 districts. Four of the districts (Sacramento 


Garden Grove Santa Ana 
San Francisco (instructional 
materials) 


Santa Ana (units 


City, San Francisco, Sanger, and Santa Ana) 
had English learner percentages higher than 
50 percent. Although student socioeconomic 
status is not addressed in this report, it is also 
noteworthy that several MiC districts had high 
percentages of low-income students, which 
may have exacerbated challenges related to 
supporting students’ mathematics achieve- 
ment.’ The percentages of low-income stu- 
dents (i.e., students eligible to receive free or 
reduced-price lunch) in MiC districts ranged 
from 51 percent (Elk Grove) to 93 percent 
(Santa Ana), with an average percentage of 
72 percent across the 10 districts. 


CCSS-M implementation timelines. Education 
reform is not immediate. Indeed, some 
researchers have argued that it may take 
several years for the impact of implementa- 
tion of education innovations to be realized 


5 Additional reporting on mathematics achievement for English learners (e.g., by language and by amount of time in 
the United States) and for other student subgroups, including low-income students and African American and Latino 


students, is forthcoming. 


6 Twenty-seven districts were invited to apply to participate in the MiC initiative, and 23 submitted an application; 10 of 
the districts that submitted an application were accepted as MiC districts, and 13 were not. 


7 Across the United States, districts with more affluent students consistently have higher math and reading achievement 


(Reardon, 2016). 
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(Fullan, 2001; Jellison, 2006; Solmon, 2003). 
Therefore, when looking at CAASPP data 
from 2015 through 2018, it is important 

to understand the different timelines for 
CCSS-M implementation that districts 
pursued throughout this period, with some 
reaching “full” implementation years before 
others. Some MiC districts began with partial 
implementation (i.e., only implementing 

the CCSS-M at selected grade levels) in the 
2011-12 school year and moved to full imple- 
mentation during the 2013-14 school year, 
while others rolled out the CCSS-M across all 
grades simultaneously (see Figure 1 on page 4 
for a visual representation of CCSS-M imple- 
mentation progress across all 10 MiC districts). 


The different implementation timelines 
across MiC districts likely affected changes 
or improvements in student achievement in 
these districts; districts that did not achieve 
full implementation until later in the initiative 
might also show improvements in student 
achievement later than other districts that 
fully implemented the CCSS-M earlier. 


The MiC districts also took a variety of 
approaches to CCSS-M implementation, with 
different indicators to define “full” implemen- 
tation and different paths to achieve it. For 
example, while adopting new instructional 
materials was a main implementation activity 
across the districts, individual districts made 
very different choices about their materials. 
Two districts organized groups of local 
educators to develop their own instructional 
materials and teaching toolkits; other partici- 
pating districts quickly adopted commercially 
produced instructional materials; and still 
other districts decided to wait several years 
before making decisions about which math 
materials to adopt. Santa Ana MiC leadership 
team leaders reported that district staff felt 
that they had achieved full implementation 
only as recently as the 2016-17 school year, 
after it had adopted instructional materials for 
all K-8 grade levels. 


Focus of this report 


This report describes three analyses that we 
conducted in order to understand student 
achievement in MiC districts, relative to 
achievement in the rest of California, on the 
CAASPP in mathematics: 


e First, it examines the percentages of 
students meeting or exceeding the 
standard over time, and compares MiC 
districts’ progress to other California 
districts’ progress on this measure. 


¢« Second, it presents regression analyses 
predicting school-level achievement 
gains between 2016 and 2018 on the 
CAASPP, with schools in MiC districts 
compared to other California schools. 


¢ Third, it provides analyses of 
student-cohort data showing how 
students in MiC districts progressed 
over three years, compared to their 
peers across the state. 


Some analyses compare scores from MiC 
districts to scores from districts that were 
invited by the S. D. Bechtel, Jr. Foundation 

to apply, and that subsequently applied, to 
participate in MiC, but that were not selected 
for participation. We chose these districts 

as a comparison group because they were 
similar in many ways, such as their student 
demographics and implementation priorities, 
to the 10 selected MiC districts. Details on the 
methodology for these analyses are included 
in Appendix A. 


We cannot draw causal links between 
specific districts’ interventions to support 
students’ mathematics achievement and 
those districts’ student outcomes. However, 
positive achievement outcomes in districts 
pursuing strategic CCSS-M implementation 
activities can shine a light on those districts’ 
practices that are worth learning more about. 


8 Several MiC districts reported that they began “full” implementation in the 2013-14 school year. One researcher 
has asserted that this year was the year that CCSS-M implementation started more broadly across the state, “only after 
previous state tests were discontinued in 2013" (Warren, 2018, p. 17). 
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Student Achievement Results 


This section presents three 
different analyses of student 
achievement. The first analysis 
presents average performance 

for MiC districts, drawing on both 
elementary and middle school 
data. Following that analysis, 

two sections describe student 
achievement results at the 
elementary school level and at the 
middle school level, respectively. 
Regression analyses in both the 
elementary and middle school 
sections describe school-level 
achievement on the CAASPP 
between 2016 and 2018; schools’ 
progress on the CAASPP in 
relation to their initial achievement 
levels; and student-cohort data 
showing how student cohorts in 
MiC districts progressed over three 
years, Compared to their peers 
across the state. 

a —  l 


Average performance of 
MiC districts, compared to 
other California districts, 
2015 vs. 2018 


¢ Summary: The percentages of stu- 
dents who met or exceeded the stan- 
dard on the CAASPP for mathematics 
achievement in 2015 and in 2018 were 
similar between MiC districts and other 
California districts. 
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As shown in Tables 1 and 2 (on page 2), 
average statewide mathematics achievement 
between 2015 and 2018 was mostly stagnant. 
However, looking only at statewide averages 
obscures information about how each of 

the more than 900 districts in the state 
performed over this same period. In order to 
show how individual MiC districts performed, 
relative to other individual California districts, 
we developed Figure 2. This graph shows the 
variation between districts’ performance in 
2015 and their performance in 2018, com- 
paring, for each district, the percentage of 
students in grades 3—8 who met or exceeded 
the standard on the CAASPP in mathematics 
in 2015 (shown on the x-axis) and the per- 
centage of students in those grades who met 
or exceeded the standard in 2018 (shown on 
the y-axis). In this graph, each district in the 
state is represented as a dot; MiC districts are 
represented as solid blue squares; and dis- 
tricts that applied for, but were not accepted 
into, the MiC initiative are represented as 
open red circles. 


The diagonal (y=x) reference line represents 
no difference between 2015 and 2018 results; 
districts whose 2015 and 2018 achievement 
falls exactly on the (y=x) reference line had 
the same percentage of students meeting 

or exceeding the standard in both years. As 
shown in Figure 2, on average, achievement 
generally follows the same pattern of per- 
formance in MiC districts as in other districts 
across the state. However, MiC districts are 
slightly more homogeneous in the percent- 
ages of students who met or exceeded the 
standard than those of the non-MiC-selected 
districts. This is indicated by the tighter clus- 
tering of the blue squares slightly left of center, 
compared to the positions of the red circles. 
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Figure 2. Average Percentages of Students Who Met or Exceeded the Standard on 
the CAASPP in Mathematics in California Districts, 2015 vs. 2018 
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Source: Authors’ analysis based on publicly available data from the California Department of Education. 


Elementary school level 


Student achievement results in 

elementary schools in MiC districts, 

compared to other districts, 2016-2018 

¢ Summary: Relative to elementary 

schools in other California districts, 
increasing percentages of MiC 
elementary schools exceeded the 
predicted achievement on the CAASPP 
between 2016 and 2018. MiC district 
elementary schools’ performance 
increased by 7 percentage points over 
the two-year period, a statistically 
significant increase. 


The district-level achievement data shown 

in Figure 2 only tell part of the story. District- 
level achievement patterns alone cannot 

tell us about how schools within individual 
districts are performing in relation to one 
another. Because conditions can vary greatly 
across schools within a district — from 


differing student demographics, to levels 
of district investment, to patterns in staff 
experience levels — it is important to learn 
which schools are achieving under which 
conditions, in order to better understand 
which practices from those schools can be 
effectively spread to other sites. 


Although MiC was a district-level initiative 
focused on improving achievement in all 
schools in a district, we believed that looking 
at individual sites would provide a fuller 
picture of progress. If the initiative was having 
an effect on individual schools, we would 
expect to see differences in average school 
performance between MiC district schools 
and non-MiC district schools with similar 
characteristics. 


One way to analyze how schools are per- 
forming in relation to one another is by 
using a regression analysis, in which we 
employ a predictive model based on data 
from all schools in the state. This model is 


\ \ \ Student Achievement Results 


Table 3. Predicted School-Level CAASPP Mathematics Outcomes of Elementary 
Schools in MiC Districts, Compared to Elementary Schools in Non-MiC Districts, 


2016-2018 
Year Number of Number of Percentage of 
Elementary Schools Schools 
Schools Performing Performing 
Beyond Beyond 
Prediction Prediction 
2016 10 MiC districts 379 182 48 
2016 13 non-MiC-selected districts 658 325 49 
2016 Remaining 575 districts in CA 3501 1624 46 
2017 10 MiC districts 378 199 53 
2017 13 non-MiC-selected districts 657 297 45 
2017 Remaining 574 districts in CA 3495 1687 48 
2018 10 MiC districts 378 207 55 
2018 13 non-MiC-selected districts 657 312 47 
2018 Remaining 574 districts in CA 3493 1691 48 


Source: Authors’ analysis based on publicly available data from the California Department of Education. 


strengthened by the breadth of available data 
from across California, which gives us a more 
complete understanding of how mathematics 
achievement is changing under the new 
standards than we would have gained from 
examining the 10 MiC districts alone. 


Using statewide data, we utilized a regression 
model to predict school achievement, based 
on prior performance and on school-level 
characteristics associated with math achieve- 
ment as measured by the CAASPP. (See 

the School-level profile regression analysis 
section of Appendix A for more information 
on how this regression model was set up.) 


Table 3 summarizes the results of this 
school-level profile analysis for elementary 
schools in the 10 MiC districts, compared to 
elementary schools in the 13 districts that 
applied for, but were not selected for, MiC 
participation and to elementary schools in all 
of the remaining California districts. The table 


shows that between 2016 and 2018,° schools 
in MiC districts made significantly more 
progress, compared to schools in the two 
other comparison groups, in exceeding their 
predicted performance; the percentage of 
schools exceeding the predicted achieve- 
ment between 2016 and 2018 was signifi- 
cantly higher (p < .05) in the 10 MiC districts 
than in the other districts. 


Table 3 shows that in 2016, across all 

three groups, about the same percentage 

of schools exceeded our performance 
prediction (48 percent of schools in MiC 
districts, compared to 49 percent of 
non-MiC-selected districts and 46 percent 
of schools in the remaining districts). In 2017, 
53 percent of MiC district schools exceeded 
prediction, while about 45 percent of schools 
in non-MiC-selected districts and 48 percent 
of schools in the remaining districts did. In 
2018, the difference was larger between MiC 


9 Although the previous achievement analyses begin in 2015, the two cohort progress analyses in this report begin in 
2016, because the 2015 achievement information is used in the regression models for subsequent years. 
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Figure 3. Achievement over Time for Elementary Schools in MiC Districts at Four 
Baseline (2015) CAASPP Mathematics Achievement Levels, 2016-2018 
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district schools and other schools across compared to elementary schools in other 


the state, with 55 percent of schools in MiC 
districts exceeding the performance predic- 
tion, compared to 47 percent of schools in 
non-MiC-selected districts and 48 percent of 
schools in the remaining districts. 


School performance: Progress in MiC 

elementary schools across different 

achievement levels, 2016-2018 

¢ Summary: Across four identified 

achievement levels, the greatest 
percentage of MiC district elementary 
schools that performed better than 
predicted on the CAASPP between 
2016 and 2018 was in the middle of 
the achievement range, although MiC 
districts also succeeded in making 
progress with their lowest-performing 
elementary schools. 


The evidence of stronger-than-predicted 
achievement in MiC elementary schools, 


districts, made us wonder about which group 
of schools in the MiC districts was most 
likely to perform better than predicted on the 
CAASPP in mathematics in each year of the 
CAASPP administration: higher-performing 
sites, lower-performing sites, or those in 

the middle. 


To answer this question, we divided the MiC 
school sample into four achievement levels?® 
based on the percentage of each school’s 
students who met or exceeded the standard 
in the CAASPP baseline year (2015). We then 
examined whether these schools performed 
beyond prediction in each subsequent year 
of CAASPP administration from 2016 to 2018. 
As shown in Figure 3, across all three subse- 
quent years, higher percentages of schools 
at the two middle achievement levels than at 
level 1 or level 4 performed beyond predic- 
tion; for both of the middle achievement 
levels, the percentage of schools performing 


10 The levels were defined as follows: level 1: less than or equal to 20 percent of students meeting or exceeding the 
standard; level 2: between 20 and 40 percent of students meeting or exceeding the standard; level 3: between 40 and 
60 percent of students meeting or exceeding the standard; level 4: greater than or equal to 60 percent of students 


meeting or exceeding the standard. 
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beyond the prediction was 49 percent or 
greater in each year. 


Figure 3 also shows that a higher percentage 
of lower-performing schools (schools at 
level 1) in 2015 performed beyond prediction 
in 2018 than in the earlier years: 47 percent 
of lower-performing schools scored beyond 
prediction in 2018, while only 40 percent of 
these schools did so in 2017 and 36 per- 
cent did so in 2016." This finding suggests 
that the lower-performing schools in MiC 
districts made increasingly greater progress 
as their implementation of the CCSS-M 
deepened between 2016 and 2018 (a change 
of 11 percentage points between 2016 and 
2018, compared to, for instance, a change of 
6 percentage points for level 3 schools during 
the same period). 


Cohort analysis describing achievement 
for all elementary school students 
« Summary: Patterns of math achieve- 

ment from 2015 to 2017 for the MiC 
elementary school cohort generally 
followed the statewide pattern, 
although, across all three years, stu- 
dents in four MiC districts (Elk Grove, 
Garden Grove, Long Beach, and San 
Francisco) had higher annual gains in 
mathematics achievement than the 
statewide average. 


Schools with large percentages of high- 
poverty students — like many schools in the 
MiC districts!? — can also have higher student 
mobility rates, especially in urban areas 
(Sparks, 2016), with students moving fre- 
quently between schools in a district or out 
of the district altogether. Because the popu- 
lation of students within a school is not stable 
from year to year, using school-level data to 


10 


examine achievement can mask details about 
individual students’ achievement patterns. 


In order to gain a clearer understanding of 
achievement in MiC districts, we felt that it 
was particularly important not only to look 
at school-level achievement, but to follow 

a cohort of students and understand those 
students’ mathematics achievement across 
their years of schooling, using student-level 
data shared by our MiC district partners. 
Specifically, we gathered individual student 
math CAASPP data from 2015, 2016, and 2017 
from nine’ MiC districts. (Student-level data 
from the 2018 CAASPP were not available for 
analysis at the time this report was prepared.) 
The data were linked so that we could 
examine individual student achievement 
across years and then aggregate achievement 
patterns for individual students to the district 
level for a cohort analysis. The MiC student 
cohort scores were then compared to scores 
for similar statewide grade-level cohorts, 
using publicly available data.“ 


As shown in Table 4, the pattern of student 
achievement for the elementary school 
cohorts in nine MiC districts generally reflects 
the statewide growth pattern (as shown in 
the first table row), with scale-score gains for 
students as they move across grades. Student 
achievement for the elementary school 
cohort across 2015-2017 was consistently 
higher than the state average in four of nine 
districts (Garden Grove, Elk Grove, Long 
Beach, and San Francisco). That these districts 
had stronger-than-state-average achieve- 
ment in 2015, the CAASPP baseline year, 

may indicate that efforts to support students’ 
conceptual understanding of mathematics in 
these districts began before the baseline year. 
For example, as shown in Figure 1 (page 4), 
San Francisco began building awareness of 


11 This trend also seems to be the case for schools at level 4 (48 percent of these higher-performing schools scored 
beyond prediction in 2018, while only 36 percent of these schools did so in 2016), although the number of schools at 
this level is small enough across each year to make such a summary statement less straightforward. 


12 As stated earlier, across all 10 MiC districts, an average of 72 percent of students were eligible for free or reduced- 


price lunch. 


13 One MiC district, Sacramento City, did not provide student-level data for the student cohort analyses. 


14 We were unable to obtain comparable student-level data from non-MiC districts. As such, state data represent data 
on the grade-level cohort in a given school year (e.g., grade 3 in 2015, grade 4 in 2016, grade 5 in 2017). 
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Table 4. Average District CAASPP Mathematics Achievement and Annual Percentage Gains for the 
Elementary School Cohort, 2015-2017 


Location Estimated Estimated Estimated Percentage Percentage 
Average Scale Average Scale Average Scale Gain Toward Gain Toward 
Score (and Score (and Score (and Next CAASPP Next CAASPP 
Corresponding | Corresponding | Corresponding | Achievement Achievement 
Achievement Achievement Achievement Level from 2015 | Level from 2016 
Level), 2015, Level), 2016, Level), 2017, to 2016 to 2017 
Grade 3 Grade 4 Grade 5 
Statewide* 2415.1 (2) 2460.5 (2) 2485.8 (2) 65% 37% 
Dinuba 2385.1 (2) 2440.1 (2) 2475.0 (2) 55% (—) 40% (*) 
Elk Grove 2432.8 (2) 2482.0 (2) 2504.6 (2) 94% (+) 49% (+) 
Garden Grove 2423.0 (2) 2477.3 (2) 2500.9 (2) 88% (+) 47% (+) 
Long Beach 2419.0 (2) 2464.2 (2) 2496.1 (2) 69% (*) 50% (+) 
Oakland 2400.3 (2) 2440.2 (2) 2471.5 (2) 47% (=) 36% (*) 
Oceanside 2384.8 (2) 2437.9 (2) 2476.3 (2) 53% (—) 43% (+) 
San Francisco 2445.1 (3) 2483.1 (2) 2510.9 (2) 95% (+) 62% (+) 
Sanger 2421.5 (2) 2462.6 (2) 2483.3 (2) 65% (%) 32% (%) 
Santa Ana 2396.2 (2) 2443.5 (2) 2465.0 (2) 53% (—) 25% (—) 


*Statewide values are actual, not estimated. 


+ Percentage gain is higher than state for the period 


*® Percentage gain is same as state for the period (within a 5% difference) 


= Percentage gain is lower than state for the period 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


the CCSS-M as early as the 2011-12 school 
year, and it fully implemented the districtwide 
Math Core Curriculum beginning in the 
2014-15 school year, the same year as the 
baseline CAASPP test. These efforts may have 
contributed to San Francisco students having 
higher overall performance that year, relative 
to their peers across the state. 


Student achievement was consistently 

lower than the state average in four districts 
(Dinuba, Oakland, Oceanside, and Santa Ana). 
In Sanger, achievement was higher than the 
state average in 2015, but was slightly lower 
than the state average in 2017. 


In addition to estimated average scale score, 
Table 4 includes information about the 

nine districts’ CAASPP achievement levels, 
shown in parentheses after the estimated 
average scale score each year. CAASPP 
scores are generally understood using these 
four achievement levels: 1 = standard not 
met; 2 = standard nearly met; 3 = standard 
met; 4 = standard exceeded (California 
Department of Education, 2018c).*° Districts, 
schools, and students may make significant 
score gains from year to year while remaining 
at the same achievement level. We also 
wanted to understand progress within the 
four achievement levels, so we calculated the 


15 We use different definitions of achievement levels for the school-profile analyses, as indicated in Appendix A. 
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percentage gains made, in each district and 
statewide, toward the next achievement level 
in each year. Percentage gains in CAASPP 
scores from 2015 to 2016 and from 2016 to 
2017 are shown in the last two columns of 
Table 4. A percentage gain of 100 or above 
means that in the following year the average 
district performance moved up to the next 
achievement level. A percentage gain of less 
than 100 means that the district performance 
moved toward, but did not reach, the next 
achievement level (see the Student cohort 
analyses section of Appendix A for more 
information about the analysis of these 
percentage gains). 


Table 4 includes symbols to visually highlight 
how the annual percentage gain for each 
district compares to the statewide per- 
centage gain for the same year."* A green plus 
sign indicates that growth for the district was 
greater than the state average (i.e., a higher 
percentage gain from year to year), a red 
minus sign indicates that growth was lower 
than the state, and a yellow star indicates that 
the district's growth was about the same as 
statewide growth. This information illustrates 
which of the MiC districts’ student cohorts 
started off with strong performance (as 
indicated by a green plus sign from 2015 to 
2016) and remained strong (as indicated by 

a green plus sign from 2016 to 2017) relative 
to the state (i.e., Elk Grove, Garden Grove, 
and San Francisco), and which gained more 
in the second year than in the first year, as 
indicated by a sign change (i.e., Long Beach 
and Oceanside). 
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Elementary school student cohort 

analysis: Achievement for English 

learners 

« Summary: Across the state, achieve- 

ment gaps between English learners 
and English speakers” are increasing. 
Although achievement in elementary 
schools in MiC districts is lower for 
English learners than for English 
speakers, English learners in most of 
the MiC districts showed stronger 
score gains than their statewide peers 
from 2015 to 2017. 


Table 5 shows achievement data for English 
learners in the elementary school cohort for 
the nine MiC districts and statewide from 
2015 to 2017. Comparing the average scale 
scores and their corresponding achievement 
levels in Table 5 to the scores in Table 4 
shows that achievement for English learners 
in elementary schools was lower than for 
students overall, both in the MiC districts and 
across the state. 


Statewide, the majority of English learners 
(87 percent) were below or far below 
standard on the 2018 CAASPP (California 
Department of Education, 2018). The disap- 
pointing statewide achievement for English 
learners in the first few years of the CAASPP 
has prompted districts across the state to 
focus on supporting their English learners 
more strategically (Ruffalo, 2018). 


Although English learners in California are 
performing more poorly than their English- 
speaking peers, there are some bright spots 
in the data from the MiC districts. Across the 
three years of the analysis, achievement for 
English learners in elementary schools in two 


16 In this and subsequent similar tables, the percentage gain for each cohort for the 2015 to 2016 period was computed 
as follows: Percentage gain = scale score change / (the cutoff of the next achievement level — the estimated average 
scale score in 2015). The percentage gain equation is similar from 2016 to 2017, except for using the estimated average 


scale score in 2016. 


17 In this section and in the comparable middle school section, English learners are defined as students who have 
been identified by their districts as “English learner” (EL) students or as students with “limited English proficiency” (LEP). 
The English speaker group may include students who are not EL/LEP students or who are “English only” students, 
depending on how they have been identified by their districts. Since we do not have the true student cohort data (i.e., 
individual student data across years) for the statewide comparison, for the statewide dataset we have used the “English 
only" group to represent English speakers. Therefore, the difference between English learners and English speakers at 
the state level may represent the upper bound of the true difference. More information about the English learner and 


English speaker categories is provided in Appendix A. 
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Table 5. Average District CAASPP Mathematics Achievement for English Learners in the Elementary 
School Cohort, 2015-2017 


Location Estimated Estimated Estimated Percentage Percentage 
Average Scale Average Scale Average Scale Gain Toward Gain Toward 
Score (and Score (and Score (and Next CAASPP Next CAASPP 
Corresponding | Corresponding | Corresponding | Achievement Achievement 
Achievement Achievement Achievement Level from 2015 | Level from 2016 
Level), 2015, Level), 2016, Level), 2017, to 2016 to 2017 
Grade 3 Grade 4 Grade 5 
Statewide* 2383.2 (2) 2412.9 (2) 2421.9 (1) 30% 10% 
Dinuba 2344.1 (1) 2396.7 (1) 2426.8 (1) 80% (+) 50% (+) 
Elk Grove 2380.6 (1) 2422.1 (2) 2433.0 (1) 140% (+) 10% (%*) 
Garden Grove 2380.1 (1) 2429.4 (2) 2446.5 (1) 160% (+) 20% (+) 
Long Beach 2388.6 (2) 2428.9 (2) 2458.4 (2) 40% (+) 30% (+) 
Oakland 2353.0 (1) 2390.8 (1) 2415.6 (1) 70% (+) 40% (+) 
Oceanside 2333.1 (1) 2379.3 (1) 2431.0 (1) 60% (+) 70% (+) 
San Francisco 2400.8 (2) 2435.3 (2) 2455.5 (2) 40% (+) 20% (+) 
Sanger 2365.3 (1) 2404.3 (1) 2426.6 (1) 90% (+) 40% (+) 
Santa Ana 2352.1 (1) 2395.4 (1) 2413.4 (1) 70% (+) 30% (+) 


*Statewide values are actual, not estimated. 
+ Percentage gain is higher than state for the period 


% Percentage gain is same as state for the period (within a 5% difference) 


= Percentage gain is lower than state for the period 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


MiC districts, Long Beach and San Francisco, 
was at the same level as or higher than 

the statewide average for English learners. 
Overall, achievement for English learners 

in the MiC districts improved from 2015 to 
2017, although achievement for this group of 
students was lower than the state average in 
Oakland and Santa Ana. 


Achievement levels for many of the MiC 
districts remained at the lowest level (1) across 
multiple years. However, as indicated by the 
many green plus signs in Table 5, the annual 
percentage gains in average CAASPP scores 
for English learners in all MiC districts were 
greater than the statewide percentage gain (as 
shown in the first row of the table), with only 


one exception: from 2016 to 2017, Elk Grove’'s 
percentage gain was the same as the statewide 
gain. This tells us that English learners in the 
MiC districts were making progress even 
though their average scores were still in the 
lowest CAASPP achievement level score range. 


Table 6 on page 14 shows the estimated 
average scale score differences between 
English learners and English speakers in each 
year from 2015 to 2017, for each district and 
(as shown in the first row of the table) for the 
state. 


Table 6 also identifies, for each of the districts 
and for the state, the slope of the score 

gap between English learners and English 
speakers, as well as identifying whether the 
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Table 6. Achievement Gap on the CAASPP in Mathematics Between English Learners and English 
Speakers in the Elementary School Cohort, 2015-2017 


Location Estimated Estimated Estimated Slope of Slope of Slope Trend 
Average Average Average Score Gap Score Gap in Gap 
Scale Score Scale Score Scale Score Between Between Between 
Difference Difference Difference English English English 
Between Between Between Learners Learners Learners 
English English English and English | andEnglish | and English 
Learners Learners Learners Speakers, Speakers, Speakers, 
and English and English and English 2015-2016 | 2016-2017 | 2015-2017 
Speakers, 2015 | Speakers, 2016 | Speakers, 2017 
Statewide 41.5 57.8 74.7 0.39 0.29 Increasing 
Dinuba 65.4 69.4 76.8 0.06 Oxi Increasing 
Elk Grove 59.0 67.6 80.8 Ons 0.19 Increasing 
Garden Grove 59.8 66.7 75.7 0.11 0.14 Increasing 
Long Beach 40.8 47.4 50.7 0.16 0.07 Increasing 
Oakland 64.6 67.4 76.3 0.04 0.13 Increasing 
Oceanside 61.2 69.3 53.6 ONS -0.23 Decreasing 
San Francisco 60.4 65.1 75.5 0.08 0.16 Increasing 
Sanger 64.9 67.4 65.5 0.04 -0.03 Increasing 
Santa Ana 65.8 71.7 77.1 0.09 0.08 Increasing 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


trend in slope of score gap between these 
groups was increasing or decreasing over 
time (see Appendix A for more information 


about these slope analyses). A slope with 
a positive number indicates that the score 


gap between English learners and English 
speakers increased from one year to the 
next, with a higher number indicating a 
relatively larger gap between these groups 
over the year. (For example, the statewide 
gap between these two student groups 
between 2015 and 2016 was 0.39, while in 


Middle school level 


MiC district as well. In Oceanside, the gap 
between English learners and their English- 
speaking peers closed over the three years. 


The analyses for the middle school cohort 
include far fewer schools than the ele- 
mentary school cohort. Therefore, the 
percentage values in this section may be 
more easily influenced by small numbers and 
should be interpreted with greater caution. 


Dinuba, the gap between the two groups was 
much lower, at 0.06.) A slope with a negative 
number indicates that the gap between these 
groups decreased from one year to the next. 


Student achievement results in middle 
schools in MiC districts, compared to 
other districts, 2016-2018 
¢ Summary: In both 2016 and 2017, 
schools in MiC districts performed 
better than predicted, compared to 


Across the state, the achievement gap between 
these groups increased during the three-year 
period, and this pattern of an increasing gap 
across all three years is evident in all but one 
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Table 7. Predicted School-Level CAASPP Mathematics Outcomes of Middle Schools 
in MiC Districts, Compared to Schools in Non-MiC Districts, 2016-2018 


Year Number Number of Percentage of 
of Middle Schools schools 
Schools Performing Performing 
Beyond Beyond 
Prediction Prediction 
2016 10 MiC districts 80 45 56 
2016 12 non-MiC-selected districts 136 55 40 
2016 Remaining 411 districts in CA 902 422 47 
2017 10 MiC districts 82 48 59 
2017 12 non-MiC-selected districts 136 54 40 
2017 Remaining 410 districts in CA 899 415 46 
2018 10 MiC districts 82 41 50 
2018 12 non-MiC-selected districts 134 66 49 
2018 Remaining 409 districts in CA 897 439 49 


Notes: This table summarizes findings from school-level profile analyses using a regression model that the authors 
developed (see the School-level profile regression analysis section of Appendix A for more information). The analyses 
summarized in this table include only 12 non-MiC-selected districts, rather than 13 as shown in Table 3, because one 


district does not have middle schools. 


Source: Authors’ analysis based on publicly available data from the California Department of Education. 


a group of similar districts that were 

not selected for MiC funding. In 2018, 
schools in MiC districts performed about 
the same, relative to the prediction, as 
schools in the comparison group and as 
schools in all California districts. 


Table 7 summarizes the data from our regres- 
sion analysis for middle schools, comparing 
schools in MiC districts to schools in a com- 
parison group of schools in similar districts 
that were not selected for MiC funding, as 
well as to schools in other districts across 

the state, based on how each group of 
schools performed relative to predictions. 
(For this middle school analysis, we used only 
12 non-MiC-selected districts, rather than 

13 as shown in Table 3, because one district 
does not have middle schools.) 


In both 2016 and 2017, a higher percentage 
of middle schools in MiC districts performed 
better than predicted, compared to middle 


schools in the group of similar districts that 
were not selected for MiC funding; the differ- 
ences are statistically significant at the p < .05 
level. In 2018, however, middle schools in MiC 
districts performed about the same, relative 
to the prediction, as middle schools in other 
districts did. Between 2016 and 2018, the per- 
centage of middle schools in MiC districts that 
were performing beyond prediction decreased 
(56 percent in 2016 versus 50 percent in 2018); 
this difference is not statistically significant. 
On the other hand, middle schools in the 
non-MiC-selected districts showed an oppo- 
site trend of improvement, with 40 percent of 
middle schools performing beyond prediction 
in 2016 and 49 percent of middle schools 
performing beyond prediction in 2018. This 
increase of 9 percentage points is significant 
at the p < .05 level. In the remaining districts 
across the state, the percentage of middle 
schools performing beyond prediction over 
the three years of the analysis remained about 
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Figure 4. Achievement over Time for Middle Schools in MiC Districts at Four 
Baseline (2015) CAASPP Mathematics Achievement Levels, 2016-2018 
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the same: 47 percent in 2016, 46 percent in 
2017, and 49 percent in 2018. Overall, the 
regression analysis shows that in MiC middle 
schools, improvement has been uneven. 


School performance: Progress in 

MiC middle schools across different 

achievement levels, 2016-2018 

¢ Summary: Across four identified 

achievement levels, the greatest 
percentage of MiC district middle 
schools that performed better than 
predicted on the CAASPP between 
2016 and 2018 was in the middle of the 
achievement range. Overall, MiC dis- 
tricts made less progress in improving 
their CAASPP scores than predicted 
for middle schools as implementation 
continued from 2016 to 2018. 


As shown in Figure 4, the greatest percentage 
of MiC district middle schools that performed 
better than predicted on the CAASPP between 


2016 and 2018 was in the middle of the 
achievement range.*® Across all three years, 
53 percent or more of the schools at the two 
middle achievement levels (levels 2 and 3) 
performed beyond prediction, but fewer than 
half of the lowest-performing middle schools 
(level 1) performed beyond the prediction. 

A larger percentage of these schools in 

level 1 did so in 2016 (48 percent) than in the 
subsequent years (39 percent in 2017 and 

42 percent in 2018). In fact, across achieve- 
ment levels 1-3, middle schools were more 
likely to perform better than predicted in 2017 
than in 2018. This information is consistent 
with the data in Table 7 showing the declining 
percentage of schools performing beyond 
prediction over the period from 2016 to 2018. 


Cohort analysis describing achievement 
for all middle school students 
« Summary: Patterns of math achieve- 
ment from 2015 to 2017 for the MiC 
middle school cohort generally 


18 The levels were defined as follows: level 1: less than or equal to 20 percent of students meeting or exceeding the 
standard; level 2: between 20 and 40 percent of students meeting or exceeding the standard; level 3: between 40 and 
60 percent of students meeting or exceeding the standard; level 4: greater than or equal to 60 percent of students 
meeting or exceeding the standard. Since only 5 schools fallin level 4, we did not interpret any findings at that level. 
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Table 8. Average District CAASPP Mathematics Achievement and Annual Percentage Gains for the 
Middle School Cohort, 2015-2017 


Estimated Estimated Estimated Darcentace Percentage 

Average Scale Average Scale Average Scale Gain ait a Gain wetne a 

Score (and Score (and Score (and Next CAASPP Next CAASPP 
Location Corresponding | Corresponding | Corresponding Achievement Achiavernent 

Achievement Achievement Achievement 

Level from 2015 | Level from 2016 

Level), 2015, Level), 2016, Level), 2017, to 2016 to 2017 

Grade 6 Grade 7 Grade 8 
Statewide* 2504.4 (2) 2525.0 (2) 2540.2 (2) 33% 25% 
Dinuba 2472.7 (2) 2480.5 (1) 2480.2 (1) 8% (—) -1% (—) 
Elk Grove 2532.7 (2) 2547.7 (2) 2560.9 (2) 44% (+) 35% (+) 
Garden Grove 2527.7 (2) 2549.9 (2) 2582.8 (2) 57% (+) 91% (+) 
Long Beach 2499.8 (2) 2523.7 (2) 2546.2 (2) 36% (*) 36% (+) 
Oakland 2462.5 (1) 2484.8 (2) 2496.3 (1) 104% (+) 11% (—) 
Oceanside 2504.4 (2) 2550.6 (2) 2556.4 (2) 74% (+) 16% (—) 
San Francisco 2531.6 (2) 2565.8 (2) 2588.0 (3) 96% (+) 110% (+) 
Sanger 2504.3 (2) 2531.5 (2) 2564.7 (2) 43% (+) 61% (+) 
Santa Ana 2459.4 (1) 2486.2 (2) 2501.4 (1) 109% (+) 15% (—) 


*Statewide values are actual, not estimated. 

+ Percentage gain is higher than state for the period 

® Percentage gain is same as state for the period (within a 5% difference) 
= Percentage gain is lower than state for the period 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


Oceanside, and San Francisco). In Long 
Beach and Sanger, average scores for the 
middle school cohort started at or lower 
than state average, but were above state 
average by 2017. Average scores for the 
middle school cohort were consistently 
lower than the state average in the three 
other MiC districts (Dinuba, Oakland, and 
Santa Ana). Of these nine MiC districts, 
Dinuba’s student achievement over these 
years was flattest; overall, the district's 
student scale scores never rose more than 
10 points across the three years.” 


followed the statewide pattern, with 
variability across nine MiC districts. 
Students in most MiC districts had 
higher annual gains in math achieve- 
ment than the statewide average in at 
least one year-over-year period. 


As shown in Table 8, across years, the 
average scale scores for the middle school 
cohort (following individual students as they 
moved from grade 6 to grade 7 to grade 8 
between 2015 and 2017) were consistently 
at or higher than the state average in four 
of nine districts (Elk Grove, Garden Grove, 


19 For many students, the transition from elementary to middle school can be a difficult transition that negatively 
influences student achievement (West & Schwerdt, 2012; Rockoff & Lockwood, 2010). In Dinuba, Elk Grove, Garden 
Grove, and Sacramento City, grade 6 is included at the elementary level, and students’ transitions from elementary to 
middle school may be a factor in achievement patterns for this cohort. 
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Table 9. Average District CAASPP Mathematics Achievement for English Learners in the Middle 
School Cohort, 2015-2017 


Location Estimated Estimated Estimated Percentage Percentage 
Average Scale Average Scale Average Scale Gain Toward Gain Toward 
Score (and Score (and Score (and Next CAASPP Next CAASPP 
Corresponding | Corresponding | Corresponding | Achievement Achievement 
Achievement Achievement Achievement Level from 2015 | Level from 2016 
Level), 2015, Level), 2016, Level), 2017, to 2016 to 2017 
Grade 6 Grade 7 Grade 8 
Statewide* 2420.6 (1) 2423.3 (1) 2427.6 (1) 0% 10% 
Dinuba 2402.0 (1) 2388.8 (1) 2405.4 (1) -20% (—) 10% (*) 
Elk Grove 2443.4 (1) 2447.3 (1) 2458.2 (1) 10% (+) 20% (+) 
Garden Grove 2469.5 (1) 2477.0 (1) 2496.6 (1) 50% (+) 70% (+) 
Long Beach 2447.1 (1) 2460.6 (1) 2479.3 (1) 40% (+) 40% (+) 
Oakland 2381.4 (1) 2397.4 (1) 2418.0 (1) 20% (+) 20% (+) 
Oceanside 2442.6 (1) 2481.5 (1) 2495.7 (1) 90% (+) 60% (+) 
San Francisco 2441.4 (1) 2470.8 (1) 2492.3 (1) 70% (+) 60% (+) 
Sanger 2410.2 (1) 2433.2 (1) 2449.9 (1) 30% (+) 20% (+) 
Santa Ana 2370.3 (1) 2391.5 (1) 2409.5 (1) 20% (+) 20% (+) 


*Statewide values are actual, not estimated. 

+ Percentage gain is higher than state for the period 

© Percentage gain is same as state for the period (within a 5% difference) 
= Percentage gain is lower than state for the period 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


Based on the annual achievement levels Middle school student cohort analysis: 
(shown in Table 8, following the estimated Achievement for English learners 
average scale score), and as indicated by e 
the green plus signs in Table 8, most of the 
districts also showed a larger percentage 
gain than the state (shown in the first row 

of the table), in at least one year. In par- 
ticular, although they began at CAASPP 

level 1 in 2015, Oakland and Santa Ana both 
progressed to CAASPP level 2 in 2016 (as 
indicated in Table 8 by the percentage gain 
from 2015 to 2016 being greater than 100), 
and San Francisco continued to make strong 
progress each year toward reaching the next 
performance level in 2017. 


Summary: Statewide, middle school 
English learners performed more 
poorly than their English-speaking 
peers. In some MiC districts, English 
learners consistently performed better 
than the state average for English 
learners, but in all MiC districts, English 
learners will need to improve much 
more to close the achievement gap 
between them and English speakers. 


As was true for the elementary school cohort, 
the scale scores for middle school English 
learners (as shown in Table 9) are lower than 
the comparable scores for all middle school 
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Table 10. Achievement Gap on the CAASPP in Mathematics Between English Learners and English 
Speakers in the Middle School Cohort, 2015-2017 


Location Estimated Estimated Estimated Slope of Slope of Slope Trend 
Average Average Average Score Gap Score Gap in Gap 
Scale Score Scale Score Scale Score Between Between Between 
Difference Difference Difference English English English 
Between Between Between Learners Learners Learners 
English English English and English | andEnglish | and English 
Learners Learners Learners Speakers, Speakers, Speakers, 
and English and English and English 2015-2016 | 2016-2017 | 2015-2017 
Speakers, Speakers, Speakers, 
2015 2016 2017 
Statewide 96.3 115.9 126.8 0.20 0.09 Increasing 
Dinuba 91.4 118.7 96.9 0.30 -0.18 Increasing 
Elk Grove 95.3 107.1 109.5 0.12 0.02 Increasing 
Garden Grove 74.8 93.8 111.0 0.25 0.18 Increasing 
Long Beach 65.5 78.3 83.0 0.20 0.06 Increasing 
Oakland 97.9 105.6 94.6 0.08 -0.10 Decreasing 
Oceanside 70.6 79.0 69.4 0.12 -0.12 Decreasing 
San Francisco 102.8 108.3 109.0 0.05 0.01 Increasing 
Sanger 101.9 106.5 124.3 0.05 0.17 Increasing 
Santa Ana 109.0 115.9 112.5 0.06 —0.03 Increasing 


Source: Authors’ analysis of student-level data gathered from each MiC district. 


students combined (see Table 8). Table 9 
shows that middle school English learners 
in five districts (Elk Grove, Garden Grove, 
Long Beach, Oceanside, and San Francisco) 
scored above the state average (shown in 
the first row of the table) for these students 
across all three years, while English learners 
in three districts (Dinuba, Oakland, and Santa 
Ana) scored below the state average across 
all three years. In Sanger, English learners 
in middle school scored below the state 
average in 2015, but scored above the state 
average in the following two years. 


While English learners in middle school 
consistently scored at the lowest CAASPP 
performance level (level 1) across districts 
and across years, some districts made more 
noticeable progress in achievement for this 


group than the state average. As shown 

in Table 9, the percentage gain in three 
districts progressed at least halfway toward 
the next level (level 2) between 2015 and 
2016: Oceanside (90 percent), San Francisco 
(70 percent), and Garden Grove (50 percent). 
Between 2016 and 2017, the same three 
districts continued to progress by 60 or 

70 percent toward the next CAASPP level 
(level 2). 


Table 10 shows data on the achievement 

gap between English learners and English 
speakers in the nine districts. As we did for 
the elementary school cohort (see Table 6 
on page 14), we computed the difference in 
scale scores for English learners, compared 
to their English-speaking peers, in each year, 
for each district and for the state (as shown in 


\ \ \ Student Achievement Results 


the first row of the table), and examined the 
extent to which the score gap was closing 
over time. A slope with a positive number 
indicates that the gap between English 
learners and English speakers increased from 
one year to the next, with a higher number 
indicating a relatively larger gap between 
these groups over the year. A slope with 

a negative number indicates that the gap 
between these groups decreased from one 
year to the next. 
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Across the state, the achievement gap 
between these groups increased during the 
three-year period, and the pattern was similar 
for students in most MiC districts. The pattern 
in Oakland and Oceanside was different, with 
the gap between English learners and English 
speakers decreasing minimally over the three 
years. This information suggests that most 
districts will need to improve their supports 
for English learners in order to narrow the 
gap between these students and English 
speakers. 
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Summary of Results 


CAASPP achievement at the district level CAASPP achievement at the student level 
(Figure 2): (Tables 4-6 and Tables 8-10): 


e The average achievement of the MiC 
districts between 2015 and 2018 was 
comparable to that of other California 
districts, though the MiC districts were 
more homogeneous. 


CAASPP achievement at the school level 
(Table 3, Figure 3, Table 7, Figure 4): 


e — Relative to schools in other districts in 
the state, an increasing percentage of 
MiC elementary schools performed 
better than predicted on the CAASPP 
in mathematics between 2015 and 
2018. 


e At the middle school level, the per- 
centage of schools performing better 
than predicted remained mostly the 
same for MiC district schools across 
years. 


e Elementary and middle schools at the 
middle two achievement levels in 2015 
were more likely to perform better 
than predicted on the CAASPP in sub- 
sequent years; the percentage of the 
lowest-performing elementary schools 
that performed better than predicted 
increased as districts’ implementation 
continued from 2016 to 2018. 


Analyses showed substantial variation 
in achievement across the MiC districts 
for both the elementary school and 
middle school student cohorts. 


However, every MiC district showed 
growth that was more positive than the 
state average for at least one annual 
growth period or student group (as 
indicated by the green plus signs in 
Tables 4, 5, 8, and 9). 


Achievement for English learners 

in MiC districts was lower than for 

all students in these districts, but in 
most MiC districts, this group showed 
stronger percentage gains from 2015 
to 2017 than the statewide average for 
English learners. 


For both the elementary and middle 
school cohorts in most MiC districts, 
the achievement gap between English 
learners and English speakers was 

not closing; indeed, in most of these 
districts, it was increasing. 
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Limitations 


Several factors in these analyses 
limit the conclusions that can be 
drawn from them. The primary 
limitation is the extent to which 
causal connections can be drawn 
between the MiC initiative and the 
student achievement outcomes 
described tn this report. As our 
other reports in this series demon- 
strate, although deep learning 
and change was happening itn the 
initiative (for example, in relation 
to the MiC districts supporting 
student mathematical discourse 
and using data for improvement), 
any impact of the initiative on 
student achievement Is indirect 

at best. Additionally, many other 
improvement activities and factors 
were occurring in each of the MiC 
districts at the same time as the 
initiative, which may have directly 
influenced the results summarized 
in this report. 

_——ee | 


This report does not include all analyses of 
interest. For example, results for analyses of 
low-income students and students of color 
in the MiC districts are forthcoming and 

will be made available on the MiC website 
(https://www.wested.org/project/math- 
in-common-evaluation/). Additionally, this 
report includes only three years of individual 
student-level CAASPP data. California 
policymakers hope and expect that CAASPP 
student achievement will improve over time 
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(California Department of Education, 2018b), 
so more substantial improvement may be 
expected once data from subsequent years 
of CAASPP testing are included in analyses. 
Similarly, as districts continue to deepen 

their CCSS-M implementation efforts, data 
from future testing periods may reveal 
achievement patterns that look different from 
those described in this report. 


Although the gathering of CAASPP data 
across all districts in the state provides an 
invaluable source of information on student 
achievement and progress, CAASPP data are 
only one indicator of achievement and prog- 
ress. Another report in this series discusses 
the value of using classroom observation 
data to understand CCSS-M implementa- 
tion (Chu, Perry, Reade, & Marple, 2019). 
Additionally, the evaluation team is analyzing 
student mathematics course-taking patterns 
in the MiC districts, to see what patterns and 
findings arise from these data. In at least one 
district, San Francisco, indications are that 
changes that the district made in response 
to the CCSS-M are paying dividends in terms 
of providing access for all middle school 
students to rigorous mathematics instruction 
in grades 6-8 (Schoenfeld & Boaler, 2018). 
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Conclusion 


At the conclusion of the five-year 
MiC initiative, although we might. 
have expected implementation 
lag (Fullan, 2001; Jellison, 2006; 
Solmon, 2003) following the 
adoption of the CCSS-M to have 
caused delays in improvements 
to student achievement, stu- 
dent achievement results in MiC 
districts reveal both substantial 
variation across the districts and 
several positive findings. 


There are signs that some MiC districts have 
accelerated student achievement on the 
CAASPP in mathematics somewhat more 
rapidly than the average statewide improve- 
ment pace. But even with hints of positive 
impacts on student achievement from the 
MiC initiative in some of the MiC districts, 
scores are still not at the level at which 
educators, parents, and the state would like 
them to be. 


The MiC initiative offered districts significant 
funding and support, allowing them to 
increase mathematics staffing, gain ideas 
about specific instructional shifts, and learn 
in collaboration with other districts. It is not 
yet clear whether the initiative’s work has 
built the kind of capacity and innovation, in 
the participating districts, that will ultimately 
result in more promising CAASPP results 
down the line. Our evaluation of Math in 
Common has suggested that district educa- 
tors are only beginning to clearly specify the 
kinds of mathematics instruction that they 
want to see in mathematics classrooms, so it 


may still be too early for districts and edu- 
cators to have designed and implemented 
effective delivery systems to achieve the 
desired outcomes from such instruction. 


Standards implementation is in the details of 
how districts, schools, and teachers interpret 
and pursue the goals of the CCSS-M, and 
correlating impacts on student achieve- 
ment may take a long time. Future positive 
achievement results for students in the MiC 
districts and in other California districts will 
depend on the accumulation of important 
learning and on capacity built for continuous 
improvement. Although we see signs of 
good news, we must remain patient about 
the disappointing early CAASPP results, and 
continue to support educators in persevering 
to make steady, positive changes. The words 
of one of our district colleagues echo this call 
for patience. 


“A more abstract threat to our 
CCSS-M implementation is the fact 
that there is a sizable group of people 
across the state getting impatient with 
the slow progress of the work. Instead 
of understanding that the way we 

are asking teachers to teach CCSS-M 
mathematics is more challenging and 
takes more time to become profi- 
cient, | fear critics’ voices will become 
louder. | recently participated in an 

... event and visited a classroom in 
another district. ... The lesson was 
grounded in best practices, and the 
teacher was skilled at connecting the 
students’ methods at the conclusion 
of the lesson. It wasn’t perfect, but 
the teacher and students were well 
on their way... . Yet, the discussion 
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among the administrators across 
districts was that this approach 
wouldn't work when the students got 
to algebra, and there was concern 
about why the standard algorithm 
wasn't introduced yet. Unless we in 
the field can convincingly persuade 
and answer these concerns, the 
CCSS-M and their associated instruc- 
tional shifts will suffer a backlash, and 
the pendulum will swing away from 
research-based instructional practices 
to a ‘back to basics’ movement." 

— 2017-18 Dinuba grant report 


Along with the positive achievement out- 
comes discussed in this report, we learned 
much from our use of the CAASPP achieve- 
ment data within the MiC initiative. The 
CAASPP data for the 10 MiC districts could be 
analyzed and presented in ways that sparked 
conversations within each district system 
about how to improve mathematics achieve- 
ment. We believe that this helped stake- 
holders within district systems gain clarity on 
the mathematics instruction that they were 
aiming for in classrooms, and strengthened 
conversations about whether and how the 
districts’ investments and choices were or 
weren't paying off for teachers and students. 


The MiC initiative also illustrated how using 
CAASPP results as common achievement 
measures could facilitate learning across 
districts. After careful work to build a sense of 
trust and community, MiC district teams dug 
into their student achievement data in col- 
laborative settings. This provided them with 
opportunities to understand and describe 
their own progress and to hear about others’ 
progress, while thinking together about the 
different policies, practices, and programs in 


24 


place for students across the districts, and 
about the effects of these investments. 


We believe that, if implementation happens in 
details of interpretation and action at multiple 
levels of complex district systems, impact 

on student achievement can be catalyzed 
when district leaders get better at under- 
standing, describing, and sharing complex 
and context-specific initiatives within their 
own district systems. Once district leaders 
understand and can describe what is hap- 
pening within their districts, they can enable 
others outside of their districts to more easily 
learn from the ideas they are testing. Using 
both summative student achievement data 
and classroom observation data to support 
deeper analysis of and sharing about the 
possible impacts of district improvement 
systems on student achievement and teacher 
practice could go a long way toward helping 
all California districts implement the CCSS-M 
as well as other standards, such as the Next 
Generation Science Standards. 


We wonder whether the state could increase 
the reporting options offered through 

the California Department of Education's 
CAASPP reporting portal to enable districts 
to track the progress of individual students 
over time, as we have done in our student 
cohort analysis. While districts receive their 
own student data, many do not have analytic 
support to conduct cohort analyses. Being 
able to analyze student cohort performance 
across an extended period of standards 
implementation and testing would enable 
districts to more closely understand their 
own achievement gaps, compare their 
achievement with their interventions, and 
learn from one another about what works to 
improve student achievement. 
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Appendix A. Methodology 


Three levels of analyses were con- 
ducted to examine math achieve- 
ment on the California Assessment 
of Student Performance and 
Progress (CAASPP) over time 
among the MiC districts: analyses 
at the district level, the school 
level, and the student level. Each is 
discussed in the following sections. 
SSS) 


District-level analysis 


We examined how MiC districts differed from 
other districts in terms of average district 
performance on the CAASPP in mathematics, 
by plotting the relationship between results 
from the baseline year and from the most 
recent year. The percentage of students 
meeting or exceeding the standard in 2015 
(the baseline year) was plotted against the 
percentage of students meeting or exceeding 
the standard in 2018 (the most recent year) 
among 10 MiC districts; 12 (middle school) 
and 13 (elementary school) non-MiC- 
selected districts (districts that applied for 
MiC funding, but were not selected to receive 
it); and the rest of the districts in the state. 


School-level profile 
regression analysis 


Since 2016, we have conducted a school- 
level profile regression analysis, using all 
schools of the same grade range (i.e., 


middle school level within this report. 


elementary or middle school*°) in California 
to examine whether a school meets the 
expected performance based on its per- 
formance from the prior year(s) and based 
on some school-level characteristics that 
are associated with math achievement as 
measured by the CAASPP. 


This analysis involves two steps. As the first 
step, a regression model is set up with the 
following elements: 


e Dependent variable: percentage of 
students meeting or exceeding the 
standard on the CAASPP in mathe- 
matics in the current year 


e Independent variable(s): percentage 
of students meeting or exceeding 
the standard in prior year(s) (for 2018, 
this element includes data from 2015, 
2016, and 2017) 


e Covariates: (1) percentage of stu- 
dents who are eligible for the free or 
reduced-price lunch program; (2) per- 
centage of English learners; (3) per- 
centages of students from each of four 
ethnic groups (Asian, Hispanic, African 
American, and White); and (4) per- 
centage of students with disabilities 


As the second step, the difference between 
the actual percentage of students meeting 
or exceeding the standard and the expected 
percentage is computed for each school in 
the MiC districts. This information is used 

to produce the school-level profile plot that 
shows how each school within a district 


20 In some districts, grade 6 students attend elementary schools, while in other districts, sixth graders are in middle 
school. Mathematics achievement can often drop as students move from one level of schooling to another (West & 
Schwerdt, 2012; Rockoff & Lockwood, 2010). For the purposes of comparison across districts, grade 6 is included in the 
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progresses over time and whether that 
change meets the expectation. 


The inclusion of all California schools in the 
analysis allowed us to compare the findings 
among the MiC districts, the non-MiC- 
selected districts, and the rest of the school 
districts in the state. For the purpose of 
comparison across years, we examined the 
percentage of schools that exceeded the 
prediction over time. In addition, in order 
to see whether certain schools tended to 
more often exceed the prediction than other 
schools, we divided the schools to four 
groups based on their achievement in 2015: 


¢ Level 1 schools: percentage of 
students meeting or exceeding the 
standard is less than or equal to 20 


e« — Level 2 schools: percentage of 
students meeting or exceeding the 
standard is between 21 and 40 


e« Level 3 schools: percentage of 
students meeting or exceeding the 
standard is between 41 and 60 


e« Level 4 schools: percentage of 
students meeting or exceeding the 
standard is greater than or equal to 60 


Student cohort analyses 


While the school-level profile analysis is 
helpful in using performance levels to 
examine whether a school meets expecta- 
tions, it does not yield information about the 
amount of improvement that students make 
in relation to prior years. Such information 
is helpful in understanding average district 
growth over time, even though the overall 
performance may remain at the same level 
from year to year. For this reason, we also 
conducted cohort analyses for the elemen- 
tary and middle school cohorts. 
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Our student cohort analyses included data 
from nine MiC districts that provided stu- 
dent-level CAASPP and demographic data 
from 2015 to 2017 (student-level 2018 data 
were not available at the time of this report's 
preparation). Using a growth model, we 
estimated the average growth on the CAASPP 
scale score over three years for all students 

in a cohort. Two cohorts of students were 
studied: 


« Cohort 1 (elementary): grade 3 in 2015, 
grade 4 in 2016, grade 5 in 2017 


* Cohort 2 (middle): grade 6 in 2015, 
grade 7 in 2016, grade 8 in 2017 


The growth model used to estimate the 
average growth from year to year for all 
students was set up as follows: 


e Dependent variable: student scale 
score on the CAASPP in mathematics 


e Independent variable: the “year” 
indicator (e.g., year 1: 2015) 


¢« Covariates: gender, ethnicity (each 
of four groups: African American, 
Asian, Hispanic, and White), English 
learner status, special education status, 
and poverty status (represented by 
students who are eligible for free or 
reduced-price lunch) 


An additional term is added to the model 
when estimating growth for a specific 
subgroup (annual achievement growth by 
gender, for example). This report focuses on 
the English learner subgroup,” which has 
been shown to be underperforming at the 
state level. We also examined the achieve- 
ment gap between English speakers and 
English learners over time. 


In this report, students’ English learner status 
is based on information from their final enroll- 
ment years for their respective cohorts. For 


21 We also conducted subgroup analyses for low-income students (those eligible for free or reduced-price lunch) and 
African American students, but did not present those data in this report because income data were not available from 
two districts (San Francisco and Oakland) and because the percentage of African American students was less than 

5 percent of all students in five districts (Dinuba, Garden Grove, Oceanside, Sanger, and Santa Ana). We hope to soon 


make these analyses available elsewhere. 
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example, for the elementary school cohort, 
students’ English learner status in grade 5 
was used in the analysis. This means that 
the students classified as English learners in 
our analysis would have been classified as 
an English learner since grade 3 (or earlier). 
Similarly, for the middle school cohort, 
students classified as English learners in 

our analysis would have been classified as 
an English learner since grade 6 (or earlier). 
CAASPP achievement gaps between English 
learners and English speakers in our analysis 
would be primarily for existing or long-term 
English learners. 


The state classifications for English-language 
fluency include the following categories: 
fluent English proficient and English only; 
initial fluent English proficient (IFEP); reclas- 
sified fluent English proficient (RFEP); English 
only; English learner; English learners enrolled 
in school in the United States fewer than 

12 months; English learners enrolled in school 
in the United States 12 months or more; 
ever-English learners; and to be determined 
(TBD). For the analysis that used state-level 
data, student groups were identified based on 
the “English learner” and “English only” cate- 
gories; the “English only" group is referred to 
in this report as “English speakers.” In the data 
provided by the districts, the “English speaker" 
category used in this report may include IFEP, 
RFEP, and “English only,” in comparison to 

the “English learner” group. That is, “English 
speakers” may include both native English 
speakers and non-native speakers who have 
become fluent English proficient. 


Since MiC districts tend to be heterogeneous 
in terms of student composition and achieve- 
ment, and since the districts carried out their 
work uniquely, the student-cohort growth 
analysis was conducted separately by district. 
This enabled us to understand how achieve- 
ment in each district progressed over time. 


For this cohort analysis, we were not able to 
use any comparison samples (as we did for 
the other two levels of analyses), because 


22 For example, if Year Y is 2016, Year Y-1 is 2015. 


individual student-level data were not 
available statewide. In order to help readers 
make sense of the values presented for 

each of the MiC districts, we used publicly 
available statewide data (such as the average 
scale score for students in grade 3 in 2015) to 
estimate growth over time for each cohort. 
Since we expect that most students in each 
grade remained in California when moving 
up to the next grade level, the resulting 
values shared for the state in this report were 
expected to be close to those that would be 
based on the true student cohorts if those 
data were available to us. 


Furthermore, we computed the per- 
centage gain for each growth estimate, 

to yield information about the amount of 
improvement being made and the distance 
to the next-highest CAASPP achievement 
level (1 = standard not met; 2 = standard 
nearly met; 3 = standard met; 4 = standard 
exceeded; see California Department of 
Education, 2018a). The percentage gain for 
each cohort was computed as follows: 


Percentage gain = scale score change / 
(the cutoff of the next achievement level 
— the estimated average scale score in 
2015 or 2016). 


A percentage gain of 100 or above means 
that in the following year the average district 
performance moved up to the next achieve- 
ment level. A percentage gain of less than 
100 means that the district performance 
moved toward, but did not reach, the next 
achievement level. 


Similarly, to help interpret the achievement 
gap over time between a particular subgroup 
(such as English learners in this report) and its 
counterpart (such as English speakers in this 
report), we computed a slope of score gap 
(difference) between subgroups from year to 
year, as follows: 


Slope = (gap or achievement difference 
between CAASPP subgroups in Year Y — 
gap in Year (Y—1))?2 / gap in year (Y—1) 


\\ \ Appendix A. Methodology 


A slope with a positive value indicates an increasing, remained the same, or decreased 
increasing gap (with higher values indicating over time. Again, we referenced the statewide 
larger gaps), while a negative slope indi- data to determine whether such a pattern 
cates a decreasing gap. With three years of is similar to or different from the state trend 
data, the slope information allowed us to during the same period. 


determine whether the subgroup gap kept 
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